United States Patent [i 9 ] 

Kim 



US0O61116O2A 
[ii] Patent Number: 6,111,602 
[45] Date of Patent: *Aug. 29, 2000 



[54] METHOD AND APPARATUS FOR 
INSPECTING SOLDER JOINTS 

[75] Inventor: Jong Hyung Kim, Sungnam, Rep. of 
Korea 

[73] Assignee: Samsung Electronics Co., Ltd., 
Suwon, Rep. of Korea 

[ * ] Notice: This patent issued on a continued pros- 
ecution application filed under 37 CFR 
1.53(d), and is subject to the twenty year 
patent term provisions of 35 U.S.C. 
154(a)(2). 

[21] Appl. No.: 08/769,522 

[22] Filed: Dec. 19, 1996 

[30] Foreign Application Priority Data 

Dec. 28, 1995 [KR] Rep. of Korea 95-61661 

[51] Int. CI. 7 H04N 7/18 

[52] U.S. Ci 348/92; 382/145 

[58] Field of Search 348/92, 95, 88, 

348/87, 145, 90, 126; 382/145, 131, 156, 
150, 144, 147, 157; 228/102 

[56] References Cited 

U.S. PATENT DOCUMENTS 

5,064,291 11/1991 Reiser 348/131 

5,293,178 3/1994 Kobayashi 348/92 

5,365,084 11/1994 Cochran et al 348/88 

5,377,278 12/1994 Ichikawa 348/92 



5,542,600 8/1996 Kohayashi et al 228/102 

5,699,449 12/1997 Javadi 382/156 

5,751,910 5/1998 Bryant et al 382/145 

6,023,663 2/2000 Kim 382/150 

Primary Examiner — Chris S. Kclley 

Assistant Examiner — Gims Philippe 

Attorney, Agent , or Firm — Burns, Doane, Swecker & 

Mathis, L.L.P. 

[57] ABSTRACT 

A method and apparatus for inspecting solder joints are 
provided. A method for inspect ing test solder joints i ncludes 
the steps of illuminating the test solder joints on a printed 
circuit board to obtain sample images of the test solder joints 
by grabbing reflected light with a charge coupled device 
(CCD) camera, classifying the sample images by an inspec- 
tor into a number of classes according to the solderin g 
quality, inputting a specin^TsHimple 1m"a^e"r5elonging to each 
class to a neural network to divide the class into a prede- 
termined n uriiher of duster s^dfetepmining a synaptic weight 
of each class bv learning each clus ter, determining a con- 
firfnetHsyriaptlcr weight by adjusting the synaptic weight 
according to a boundary condition between the clusters 
belonging to neighboring different rJassesyAnd selecting a 
similar cluster by comparing the similarities between the 
o utputs oft he neural network with respect to the test solder 
joints base^^rnmtfiraed-^flapttTweights, and each output 
of the inspector's class. The method and apparatus for 
inspecting solder joints can guarantee satisfactory classifi- 
cation accuracy with the minimum number of prototype 
images. 

3 Claims, 4 Drawing Sheets 





09/25/2003, EAST Version: 1.04.0000 



U.S. Patent 



Aug. 29, 2000 



Sheet 1 of 4 



6,111,602 



FIG. 1 



MAIN PROCESSOR 



NEURAL NETWORK CLASSIFIER 



COLOR IMAGE GRABBER 



R FRAME 



'8 



G FRAME 



B FRAME 



3 




09/25/2003, EAST Version: 1.04.0000 



U.S. Patent Aug. 29, 2000 

FIG. 



Sheet 2 of 4 



2 



6,111 



CLASSIFICATION 
OF SOLDER 
JOINT QUALITY 




TYPICAL COLOR 
PATTERNS 


1 






t 

1 

LEAD 
1 


B 

f— 1 


-J 


1 

IA 






c 

H 

LEAO 
. .. ( ., 


3 R 

If 




k 






LEAO 
1 


B 




EA 






( 

LEAO 
I 






1 

E 






( 

1 

LEAO 
1 1 


3 B R 

£3* 





09/25/2003, EAST Version: 1.04.0000 



U.S. Patent Aug. 29, 2000 Sheet 3 of 4 6,111,602 



FIG. 3 




09/25/2003, EAST Version: 1.04.0000 



U.S. Patent Aug. 29, 2000 

FIG. 



Sheet 4 of 4 

4 



6,111,602 




09/25/2003, EAST Version: 1.04.0000 



6,111,602 

1 2 

METHOD AND APPARATUS FOR sample image belonging to each class to a neural network to 

INSPECTING SOLDER JOINTS divide the class into a predetermined number of clusters; 

determining a synaptic weight of each class by learning each 

BACKGROUND OF THE INVENTION cluster; determining a confirmed synaptic weight by adjust- 

, , 5 ing the synaptic weight according to a boundary condition 

The present invention relates to a method and apparatus betW£en the clust6fS belon ^ to neighboring diffcrent 

for inspecting solder joints, and more particularly, to a cksscs . and selecting a similar class by comparing the 

method and apparatus for ins E ectm S _soJa^r_4mnlS_^a similaritics between the outputs of the neural network with 

printed circuit board (PCB) by using a circular illumination respect to tfl6 ^ sMeT joints based Qn amSnaBi synaptic 

technique and a neural network classifier. w weights and 6ach outpm of ^ inspector . s class . 

The solder joints of surface mounted components on a HerC) in the step of adjusting the synaptic weigbtSj jf the 

PCB appear m various three-dimensional (3D) shapes j t of ano(her class (akes the synaptic weigW 

according to the quanUty of soldering material and a sol- UBamA b a speciflc d it fa desirable ^ ^ synaptic 

denng temperature condiUon, etc. Moreover, the surfaces of weJ ht of the ^ fc altered in the movin 

the solder joints are specular. To judge the soldering quality 15 ^ m tbe j nput 

of the solder joints, there have been proposed various neural ', , ... . 

Detwork based methods. By these known methods, the . In 1 the fP of determining the synaptic weights, if the 

specular surfaces of the solder j oints are illuminated with a Warily between the synaptic weight of a current input and 

lamp or lamps, light reflected from the specular surface is f P«t winner synaptic weight based on a previous input is 

received in a charge coupled device (CCD) camera to ^ larger than a predetermined range, it is preferable that a new 

acquire digitized image data of the received 3D shape out P ut ncuron 15 generated. 

information of the solder joints. The acquired 3D image data The illumination apparatus includes a number of different 

is learned and classified in a neural network by an artificial colour lamps which illuminate the surface of solder joints at 

intelligence technique to recognize the shape of the solder different incident angles, wherein a single CCD camera can 

joints, and thereby determine the soldering quality of the 25 receive me reflected light created by each lamp, 

solder joints. There is also provided an apparatus for inspecting solder 

However, such a conventional neural network learning j° ints comprising: illumination means for illuminating the 

and classification method is typically based on an unsuper- solder joints; a CCD camera which captures reflected light 

vised version of Kohonen's learning vector quantization created by the illumination means; an image grabber which 

(LVQ) algorithm^ which requires a large number of pro to- 30 digitizes an image of the CCD camera into digital frame 

types to insure satisfactory classification accuracy, thereby data i and a mam processor with a neural network which 

lowering the overall efficiency of the learning and classifi- confirms the synaptic weight of each class by learning the 

cation process. In practice, self -organized clusters or self- data input from the image grabber by each class supervised 

organized prototypes generated in the neural network based by me inspector. 

learning process are just a measure forjudging the similarity 35 Here, the illumination means can simply be comprised of 

between respective test materials. As a result, the case where a plurality of circular colour lamps which have different 

the inspector's intention does not correspond to the design- incident angles. 

er ' s - Preferably, the colour lamps are of high-frequency fluo- 

Meanwhile, a conventional illumination method employs rescent lamps, which have green, red and blue colours. Also, 

a plurality of lamps and to a single camera or a plurality of 40 the colour lamps are preferably coaxially tiered with a CCD 

cameras and a single lamp, to measure the slope of the camera located axially at the top of the lamps, 
surfaces of solder joints. Here, either of the lamp(s) or the 

camera(s) is moved in order to acquire the image data. In BRIEF DESCRIPTION OF THE DRAWINGS 

such an illumination method, since various images should be ^ . . . . ... * . 

j j j 1 j *• 1 • 4< FIG. 1 is a schematic perspective view of an illumination 

recorded and analyzed every time a lamp or a camera is 4i 4 . , 4 r . r . u • • 

. J . 4 . ' ■ 4 , ^ j i_ . 1 system employed to acquire an image the solder joints in a 

moved, much processing time is not only consumed but also . . \ . ^ ^ 0 A J 

. . . . 1 solder joint inspection apparatus according to the present 

expensive equipment is required. t . J r 

r ^ r n invention. 

SUMMARY OF THE INVENTION FIG. 2 is a table illustrating classification, 3D shapes and 

50 typical colour images for various solder joints. 

Therefore, to solve the above problems, it is an object of ri „ ~ . . .„ . 1f . , , 4 

4 . , ., r tJ 7 . . . J . FIG. 3 is a view illustrating a self-organized cluster 

the present ^invention to provide a solder join inspection module &d ^ fa each isor classification class . 

method and apparatus which has enhanced classification 

accuracy using a minimum number of prototypes based on FIG 4 18 a view illustrating a winner prototype image in 

an adaptive learning vector quantization (LVQ) method. 55 a first learning vector quantization (LVQ1) phase. 

It is another object of the present invention to provide a FIG - 5 * a view illustrating a winner prototype image in 

method and an apparatus for inspecting solder joints in a a second leamin S vector quantization (LVQ2) phase, 

short processing time with low-priced equipment by using a DETAILED DESCRIPTION OF THE 

circular illumination technique. PREFERRED EMBODIMENT 

To accomplish the above object of the present invention, 60 

there is provided a method for inspecting test solder joints, An embodiment of the present invention will be described 

the method comprising the steps of: illuminating the test with reference to the drawings. 

solder joints on a printed circuit board to obtain sample FIG. 1 is a schematic perspective view of an illumination 

images of the test solder joints by capturing reflected light system employed to acquire an image of solder joints in a 

with a charge coupled device (CCD) camera; classifying the 65 solder joint inspection apparatus according to the present 

sample images by an inspector into a number of classes invention. As can be seen from FIG. 1, a lead 2 of a surface 

according to the soldering quality; inputting a specific mounted component is attached on a printed circuit board 
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(PCB) 1 by means of a solder joint 3. Three colored circular ters do not yet form accurate boundaries between each other, 

lamps 4, 5 and 6 which have different diameters, are Thus, in LVQ2, the boundaries are precisely adjusted with 

co axially tiered over the solder joint 3. The light illuminated respect to clusters formed in the other classes to insure 

from the colored lamps 4, 5 and 6 is reflected from the classification accuracy. The adjusted result is expressed as a 

specular surface of the solder joint 3 to be picked up by a 5 weight value connected with one neuron, in which the 

CCD camera 7 which is located the solder joint 3. The weight value has the same dimension as that of the input, 

colored circular lamps 4, and 6 are high-frequency fluores- which is called a prototype image. 

cent lamps which are preferably tiered in the sequence of The structure of the self-clustering module to be applied 

green (G), red (R), and blue (B) moving upward from an to each expert inspector classification class is shown in FIG. 

inspection surface. Due to the different heights and diam- 10 3. An input of each color image which is stored in the digital 

eters of the lamps, incident angles with respect to the solder memory is composed of a pixel (ij) where three colour 

joint are different. Ihus, different color images can be patterns: the red, green and blue patterns are formed. Each 

obtained depending on the slope of surface of solder joints. color pattern is divided into each color frame 12, 13 or 14 

For example, when the uppermost blue color lamp 4 having having the same magnitude as that of the input image. Thus, 

the smallest diameter illuminates the solder joint 3 at an 15 a color intensity of each pixel is used as an input value of an 

angle of 20° with respect to a vertical line of the PCB 1, the mpil t node 11. The input color image can be represented by: 

red color lamp 5 illuminates the solder joint 3 at an angle of l mini 123 t'12 n ' 1 2 m)\ (l) 

40° and the green color lamp 5 illuminates the solder joint * cMJ)r ...<-,,...,«;- m)\ ( ) 

3 at an angle of 70°, the surface of the solder joint having the where subscript c indicates the class of the input image, 

slope of an angle around 10° is seen as blue in one frame of 20 U(M ,j) means an intensity value at the (i,j) pixel in the 1th 

image which has been obtained as described above; the digital memory frame, that is, 1 represents an index of each 

surface having a gentle slope of an angle around 20° is seen color (1-red, 2-green, 3-bfue), and (n,m) indicates the size 

as red; and the surface having a steep slope of an angle of the image. One image ic composed of n =m pixels, 

around 35° is seen as green. Because such a color image The dimension of the input nodes is the same as that of the 

contains the 3D information of solder joint shapes, a real 25 input winner images. Output neurons 15 are fully connected 

shape of the solder joint can be extracted from the color to all input nodes by the synapses, A set of the synaptic 

image. weights between the kth output node and the input nodes is 

The information obtained by the CCD camera by using defined as follows: 

circular illumination is transferred to a color image grabber w c *-{w c *(;i, /)|/-i,2,3, 1-1,2, . . . , n, /-i,2, . . . , m] (2) 

8 in the form of an electrical signal and then is converted 30 ^ ^ ^ ^ ^ ^ rf ^ 

mto quantitative information of a color image composed of The m yalue of ^ k(h neufon fc a correlation 

each color frame. Hie above ^formation is stored m the ^ of a bri htness vahle of the m t ^ ^ the 

memory of a main processor 9, and classified via a learning Uc wei Q which fc e ed b 
process m a neural network classifier of the main processor 

9 35 

» m 3 (3) 

Before performing a neural network classifying process, s * ~ w < ° c ~ Yj^jYj w ^ li ' fi' lc & ^ 

a large number of colour images of solder joints are < J i 
extracted, and the qualities of the solder joints are classified 

by an expert inspector. For example, the qualities are clas- T , . 

sified into five classes: insufficient soldering (I); insufficient 40 ? *• self-clustering module, the input value of the output 

but acceptable soldering (IA)k acceptable soldering (A); node " ,ndl = ales the ™iu«l nimknly, that is, a degree of 

excess but acceptable soldering (EA); and excess soldering ^semblance between the brightness value of an input image 

(E). HG. 2 shows typical soldering joint examples of the five lnd t>* synaptic weights. As shown in FIG. 3, the output 

classes and of their colour patterns appearing in the color value . of , ^ch neuron is positively feedback to itself and 

images, where blank represents blue color, oblique line 45 negatively feedback to the other output neurons. Then, only 

represents red, and black represents green. one neuron with the maximum output value converges to a 

n i_, m \ j • • -l • as positive value, and the others converge to zero, which is a 

The blue (B) pattern appears dominantly in the insuffi- ri[)ci fc of a titive learning mechanism. In other 

cient soldering © image^ In the insufficient but acceptable wofd on] 0Qe neuron oaaoeeitA ^ me tic 

soldermg (IA), the red R) and green (G) patterns begin to 5Q wei ht whkh ^ nearesl to ^ brightnesg value of the J cur £ n , 

appear In the accep able soldermg (A) the green (G), red j ^ as a ^ ^ wd h(s 

(R) and blue (B) patterns appear sequentially. In the excess rf ^ winner m tQ be leirned resemb ,£ ^ ^ , 

but acceptable soldering (kA), red (R) pattern appears image 

dominantly in the image. The excess soldering joint (E) has Iq ^ ^ a Hcation of me LV Q algorithm, the 

a convex solder fillet, ^ and thus tie blue (B) pattern is 5S number of output Deur0 ns directly influences classification 

enclosed by the red (R) and green (G) patterns. ^ computationa i burden . If the number of output 

Such an image of the test solder joint which is classified nodes is too large, the input images can be precisely clas- 

by the expert inspector is input into a neural network, and s ifi ec i although processing time is lengthened. On the other 

subjected to a learning process. nan d 7 if the number of output neurons is too small, it is 

The algorithm according to the present invention is based 60 difficult to ensure a precise classification of the input images 

on a supervised version of LVQ (Learning Vector although processing time is shortened. 

Quantization) and has two learning phases: LVQ1 and Therefore, it is essential to add an additional mechanism 

LVQ2. In LVQ1, as mentioned above, a self-clustering such as an adaptive learning mechanism to reduce the 

module is applied to each supervised class classified by the number of output neurons while maintaining a desired 

expert inspector, to self-produce a number of clusters. Such 65 accuracy of classification. 

a self-clustering module is applied to all the classes to An adaptive learning mechanism is designed to obtain the 

produce the self -organized clusters. The self -organized clus- optimal number of the output neurons. The main idea of this 
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mechanism is that if the similarity between a newly input previous procedures equations (5) through (9) until all test 

image and a winner to which the input image belongs is less input images have been assigned. 

than a threshold, the newly input image is not made to Although the above procedures are finished, the boundary 

belong to the winner, but an additional output neuron is between the prototype images are not clear. Whenever a new 

independently produced to produce another cluster and then 5 prototype image generated, the boundaries between the new 

to make the newly input image to belong to the produced one and other existing prototype images should be adjusted, 

cluster. A similar idea has been considered by other Therefore, the above procedures are repeated until no new 

researchers, who have a great similar interest in the other prototype image is incremented during this repetition. In the 

practical application such as image compression using the same manner, such a learning process is applied in turn to all 

LVQ algorithm. 10 supervised classes, respectively. 

The detailed procedure of training or learning the overall In LVQ2 learning phase after completing the LVQ1 

neural network using an adaptive learning mechanism is learning phase, clusters are generated in the supervised 

described as follows. classes, and synaptic weights are adjusted by the sample of 

First, apply a self -clustering module to one of the super- other classes. The trained synaptic weights in the LVQ1 are 

vised classes, the cth class. Assign the colour intensity of the 15 used as the initial values of the synaptic weights for the 

first sample image to the synaptic weights of the first output LVQ2. 

neuron. When a winner is determined via a competition learning 



W>I C , £=1 (4) 



method, synaptic weights are changed only if a prototype 
image of another class is selected as the winner. For 



Take the next sample image and determine a winner 20 example, when an input I,, belongs to the qth class and a 

output neuron winner is found, if the input is misclassified into the cth 

W c ^ is a set of the synaptic weighs connected with the class > synaptic weights representative of the cth class are 

winner output neuron. W c win has the relationship of the changed as follows: 

following expressions for all k: ^ w c -(,)=w -(^(Oi^-w (io) 

y y y ivr*^/ / ') - / (/ / ')| < ^ c e( l uat i on intuitively tells us that when cth class 

tjt' C winner is found although the input image 1, belongs to the 

qth class, the cth class synaptic weights are moved in the 
A jt 30 opposite direction away from the misclassified input I , as 

LLL lw < ik > J) ~ ^ '» ^ shown in FIG. 5. 

After determining all synaptic weights through the above 
LVQ1 and LVQ2 learning phases, the synaptic weights are 
where the symbol || is an operator for calculating the registered as the confirmed synaptic weights, 
absolute difference of two scalar values. (When starting, 35 As shown in FIG. 1, images of solder joints are extracted 
there is only one output neuron.) by using illumination and a CCD camera, the images are 

Assign an input I c to W^'* if the following condition is input to a an ion system neural network of a main processor, 
met: and output values are determined based on the confirmed 

synaptic weights mentioned above. A supervised class which 
40 has an output value similar to such an output is selected, to 



\\WJ-lMx(nxmx3) (6) 



then, update the synaptic weights of the winner neuron as decidc # ^ solder Joint * assigned to a class which has the 

follows: best similar output. Accordingly, the solder Joint is assigned 

to one of the classes which are classified in advance by an 

w/** (r+i^w^'" (^(Otlc-W^C*)} (7) expert inspector, and is evaluated as having a quality cor- 

. 4 - . c r , - ^ - 45 responding to the class, 

where t is the number of iterations for learning, ri (t) is a K , 1 u j* * 4L 

j - , - , ,1 . j , . Vrn v • As described above, according to the present invention, a 

dynamic learning rate that decreases as t increases. The c is 4 , , . £ . . . , ' 

defined as the similarity bound, which determines whether m f° d ^ * l " Sp6Ctlng soldcr .J 0I ( f can 

the new input image is assigned to the winner neuron. Its fto" 

value is determined by the user. The update rule of the D ™, cr °. p . ro . ° ^? imageSi 

equation (7) can be understood in terms of the movement of 50 i * ^ ( V £m f e J . ^ r . A . . , lL . , 

the winner prototype images in the hypothesis input space, 1 met ^ od , for ^V^ng solder joints, the method 

as shown in FIG. 4. By updating the synaptic weights, the com P rism g tne ste P s 0I: 

winner prototype image moves towards the presented input illuminating solder joints on a printed circuit board and 

sample obtaining sample images of the solder joints with a 

Do not assign the input \ c and the winner neuron W c ^ if 55 charge coupled device (CCD) camera; 

the following condition is met: classifying the sample images by an inspector into a 

number of classes according to the soldering quality; 
inputting a specific sample image belonging to each class 

which means that the similarity between the input I c , and the 60 to a neural network to divide the class into a predeter- 

winner is less than a reference value. Therefore, a new mined number of clusters; 

output neuron is produced and new synaptic weights are determining a synaptic weight of each class by training 

assigned as W/**': each cluster; 

w e " w (f)«i c -Ti(/){i e -w c ,t **(Q} f r«i (9) adjusting the synaptic weight according to a boundary 

65 condition between the clusters belonging to neighbor- 
Then, the new prototype image has a decision boundary ing different classes and determining the adjusted syn- 
with the winner V/ C wtn in the hypothesis input space. Repeat aptic weight as a confirmed synaptic weight; and 



||W c ' w ' , -y^><(«xmx3) (8) 
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selecting a similar class by comparing an output of the 
neural network based on the confirmed synaptic weight 
and outputs an of inspector's class, 

wherein the step of determining the synaptic weight 
includes the steps of generating a new output neuron 5 
when the similarity between a current input and a preset 
synaptic weight based on a previous input is beyond a 
predetermined range, 

generating a new cluster when a new output neuron is 
generated, and 

causing the current input to belong to the new cluster, and 

wherein in the step of adjusting the synaptic weights, only 
when an input of one class takes the synaptic weight of 



10 



8 



another class, the synaptic weight of the latter is altered 
in a direction moving away from the input of the 
former. 

2. The method for inspecting solder joints according to 
claim 1, wherein the illumination is carried out by a plurality 
of circular lamps having different colours and coaxially 
arranged to illuminate the surface of the solder joints at 
different incident angles. 

3. The method for inspecting solder joints according to 
claim 2, wherein the CCD camera is arranged axially 
relative to the plurality of lamps. 
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